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The  significant  results  of  the  analysis  and  results  of  metrics  and  weightings 
task  were: 


a) 


b) 


c) 


d) 


e) 


f) 


Selection  and  implementation  of  the  ASD/McDonnell  Douglas  F-15A  LCOM 
computer  simulation  as  the  baseline  model  with  which  to  perform  initial 
validation  experiments  on  the  new  maintenance  metrics  and  weightings 
developed  under  this  study  contract. 

Performance  of  an  initial  series  of  baseline  and  experimental  simulation 
runs  to  establish  a  basis  for  comparison  of  the  newly  developed  metrics 
and  weightings  for  the  Phase  I  study  equipments  (engines  and  avionics) 
with  the  metrics  and  weightings  currently  used  in  the  F-15A  LCOM.  These 
simulation  experiments  tested  the  fidelity  of  the  maintenance  metrics 
predictor  equations  using  the  1977  F-15A/Bitburg  Air  Base  data  base. 

Performance  of  an  initial  difference  analysis  of  the  newly  developed 
versus  currently  used  metrics  and  weightings  based  on  the  initial  series 
of  experimental  simulations.  This  initial  analysis  indicated  that  the 
LCOM  Failure  Clock  models  developed  for  avionics  were  able  to  recreate 
baseline  simulation  conditions  within  an  acceptable  degree  of  variability. 
The  average  percent  difference  of  25  selected  critical  output  variables 
from  baseline  values  was  8.25%.  The  engine  failure  clock  model  exhibited 
a  lower  degree  of  estimation  accuracy,  however,  and  indicated  the  need 
for  further  refinement.  In  this  case,  the  average  percent  difference  of 
the  25  outputs  from  baseline  values  was  61.83%. 

Selection  and  implementation  of  the  ASD  KC-135A  LCOM  computer  simulation 
for  a  series  of  experiments  which  compared  current  maintenance  metrics 
with  the  new  metrics  for  three  different  KC-135A  bases  in  different 
geographical  locations  and  environments.  These  bases  were  Loring  AFB, 
Maine;  Seymour-Johnson  AFB,  North  Carolina;  and  Castle  AFB,  California. 

Performance  of  series  of  calibration  and  experimental  runs  for  each  of 
the  selected  KC-135A  bases.  These  experimental  series  established  simu¬ 
lated  steady-state  operation  and  maintenance  (O&M)  conditions  at  each 
base  under  equipment  failure  rates  controlled  in  turn  by  (1)  existing  ASD 
standard  failure  clock  values,  (2)  maintenance  metrics  derived  failure 
clock  values,  and  (3)  1977  actual  base-specific  data  derived  failure 
clock  values. 

Performance  of  difference  analyses  on  the  above  series  of  runs.  These 
compared  the  simulation  results  which  used  the  standard  ASD  metrics  and 
the  results  using  the  new  failure  predictor  equations  against  the  base¬ 
line  simulation  results  obtained  by  using  the  actual  1977  KC-135A  failure 
data  from  Loring,  Seymour-Johnson,  and  Castle  Air  Force  Bases.  The 
findings  are  summarized  as  follows: 

Average  percent  differnce  of  the  25  selected 
critical  output  variables  from  the  1977  baseline 
simulated  values  was  -  - 


Loring  AFB: 
Seymour-Johnson  AFB: 
Castle  AFB: 


Using  ASD  Std 
Failure  Clocks 

-  2.39% 

-  8.26% 

+  1.02% 


Using  Maint.  Metrics 
Derived  Failure  Clocks 

-  2.85% 

-  8.93% 

-  2.79% 
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f)  Continued 

Selected  output  variables  from  the  baseline  simulations  of  each  base  were 
compared  to  actual  1977  O&M  histories  from  G033B  and  D056E  data  for  these 
bases.  This  test  of  the  overall  fidelity  of  the  ASD  KC-135A  LCOM  simulation 
indicated  an  average  difference  of  approximately  10%  between  LCOM  simula¬ 
tion  results  and  1977  actuals  over  the  three  KC-135A  bases  tested.  This 
fidelity  was  very  acceptable  when  it  is  considered  that  the  ASD  standard 
simulation  resource  base  was  not  tailored  to  fit  the  conditions  at  each 
specific  base. 

^The  findings  fgom..@-and  (ffr  above  indicate  that  the  new  maintenance  metrics 
predictor  equations  can  provide  acceptable  estimations  of  overall  aircraft 
maintenance  demand  rates  under  a  wide  variety  of  equipment,  operational,  and 
environmental  characteristics.  These  general  models  could  be  used  for 
predicting  equipment  failure  rates  in  many  user  situations  such  as  LCOM 
analyses  and  new  aircraft  concept  definition _ _ 


This  document  is  the  fourth  of  a  series  of  five  seeing  Technical  Reports 
generating  from  this  study,  namely:  \ 

0194-10089-1  Development  of  Maintenance  METRICS  To  Forecast  Resource 

Demands  of  Weapon  Systems  (Phase  I  -  Analysis  and  Evaluation) 

D1 94-1 0089-2  Development  of  Maintenance  METRICS  To  Forecast  Resource 
Demands  of  Weapon  Systems  (Parameter  Prioritization) 

Dl 94-1 0089-3  Development  of  Maintenance  METRICS  To  Forecast  Resource 
Demands  of  Weapon  Systems  (Maintenance  Metrics  and 
Weightings) 

D194-10089-4  Development  of  Maintenance  METRICS  To  Forecast  Resource 

Demands  of  Weapon  Systems  (Analysis  and  Results  of  Metrics 
and  Weightings) 

Dl 94- 1 0089-5  Development  of  Maintenance  METRICS  To  Forecast  Resource 
Demands  of  Weapon  Systems  (METRICS  Final  Report) 


t>cr<' '•'•inn  For 


iC  TA’« 


Jut  • 


Axc.i :  ■  : 


_ Unclassified _ 

SECURITY  CLASSIFICATION  of  This  PAGC(IF/I»n  n*t»  htfnd) 


SUMMARY 


This  report  describes  the  results  of  the  eighth  task  of  an 
eight  task  study.  The  total  effort  is  intended  to  develop  more  accurate 
metrics  and  weightings  to  be  incorporated  into  the  Air  Force  method 
(Logistics  Composite  Model  (LCOM))  for  determining  manpower  and  other 
resource  requirements  for  operational  and  developing  weapon  systems. 

The  eight  study  tasks  comprising  this  study  were  as  follows: 


Task 

I 

Review  of  Related  Research 
(Boeing  document  D . 94-1 0089-1 ) 

Task 

II 

Select  Equipment  for  Investigation 
(Boeing  document  D1 94- 1 0089-1 ) 

Task 

III 

Identify  Parameters  for  Investigation 
(Boeing  document  D1 94-10089-1 ) 

Task 

IV 

Identify,  Obtain,  and  Integrate  Study  Data 
(Boeing  document  0194-10089-1) 

Task 

V 

Analyze  and  Prioritize  Parameters 
(Boeing  document  01 94-1 0089-2) 

Task 

VI 

Maintenance  Metrics  Development 
(Boeing  document  D1 94-1 0089-3 ) 

Task 

VII 

Maintenance  Weightings  Development 
(Boeing  document  D1 94- 10089-3) 

Task 

VIII 

Analysis  and  Modification  of  Metrics  and  Weightings 
(Boeing  document  D1 94- 1 0089-4) 

PROBLEM 


The  increased  concern  with  the  manpower  required  to  support 
weapon  systems  currently  in  operation,  as  well  as  those  in  development 
has  created  the  need  for  more  accurate  methods  of  projecting  maintenance 
requirements.  Meeting  this  need  requires  the  development  of  realistic 
measures  of  maintenance  rates  for  all  of  the  diverse  hardware  that 
makes  up  a  weapon  system.  In  addition,  the  impact  of  operations  and 
environmental  conditions  needs  to  be  identified  to  insure  the  sensitivity 
of  the  maintenance  metrics  that  are  developed. 

To  date,  the  manpower  and  other  resource  requirements  essential 
to  the  Operations  and  Support  of  a  weapon  system  have  been  determined 
using  the  traditional  "flying  hours"  and  "sortie  rate"  measures.  The 
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deficiencies  of  these  traditional  measures  are  well  known  and  such 
measures  frequently  are  found  to  be  totally  irrelevant;  for  example, 
many  avionics  items  operate  or  are  cycled  greatly  in  excess  of  the 
related  flying  hours.  These  traditional  measures  are  also  insensitive 
to  variations  in  operations  and  environmental  conditions.  The  present 
difficulties  then  lie  in  the  fact  that  the  currently  used  metrics  do  not 
consider  the  inherent  differences  between  the  individual  subsystems  of 
a  weapon  system  and  are  relatively  insensitive  to  operational  and 
environmental  conditions. 

The  problem  for  this  portion  of  the  study  was  to  plan  and 
execute  a  series  of  validation  experiments  for  the  new  metrics  and 
weightings  models  developed  during  the  preceding  study  tasks.  The 
testing  of  these  new  metrics  and  weightings  must  be  performed  in  the 
context  of  actual  operative  LCOM  simulations  if  the  validation  is  to 
have  credibility.  They  must  be  transformed  to  Failure  Clock  Values 
required  to  drive  the  Maintenance  Networks  of  selected  LCOM  simulations 
so  that  comparisons  may  be  r.de  between  these  new  metrics  and  those 
currently  in  use. 

APPROACH 


The  approach  taken  for  this  portion  of  the  study  effort  was 
to  select  and  implement  existing  baseline  LCOM  simulations  to  use  as 
vehicles  for  the  calibration  and  validation  experiments  on  the  mainte¬ 
nance  metrics  models  developed  in  preceding  study  tasks  VI  and  VII 
(see  Boeing  document  0194-10089-3).  The  initial  model  selected  was 
the  ASD/McDonnell  Douglas  LCOM  simulation  of  the  F-15A  aircraft  at 
Bitburg  Air  Base,  Germany.  A  series  of  calibration  runs  of  baseline 
model  were  executed  using  the  failure  clock  and  task  selection 
probability  standards  developed  from  the  historical  F-15A/Bitburg  data 
base.  Experimental  runs  were  then  executed  with  failure  clock  values 
based  on  the  generalized  metrics  models  developed  during  the  course 
of  this  study.  Comparisons  were  then  made  between  the  experimental 
and  calibration  simulation  results  and  an  evaluation  made  of  the  success 
of  the  new  generalized  metrics  in  receating  the  specific  historical 
conditions  portrayed  by  the  calibration  simulation  runs.  Determination 
was  then  made  of  the  acceptability  or  need  for  modification  and  refine¬ 
ment  of  the  newly  developed  metrics. 

Subsequent  series  of  experiments  were  then  planned  and 
executed  using  the  standard  ASD  KC-135A  LCOM  simulation.  This  simu¬ 
lation  was  implemented  with  the  scenarios  from  three  widely  separated 
and  environmentally  different  bases;  i.e.,  Loring  AFB,  Maine;  Castle  AFB, 
California;  and  Seymour- Johnson  AFB,  North  Carolina.  Calibration  and 
experimental  runs  were  then  made  for  each  base  first  using  the  standard 
values,  then  the  study  predicted  values  for  failure  clocks,  and  finally 
failure  clock  values  derived  from  actual  operational  and  failure  data 
from  the  three  bases.  Comparisons  were  then  made  between  the  simula¬ 
tion  results  from  the  standard  runs,  the  new  metrics  runs,  and  the 
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actual  metrics  runs.  These  sets  of  results  were  then  compared  with 
actual  historical  performance  data  from  the  subject  bases  to  determine 
which  type  of  metric  yielded  results  closest  to  actual  historic  values, 
and  to  check  the  general  fidelity  of  the  LCOM  simulation  used  in  the 
experiments. 

RESULTS 


The  foregoing  approach  was  initially  applied  to  the  mainte¬ 
nance  action  demand  estimating  models  developed  during  the  Phase  I 
portion  of  the  study  for  aircraft  propulsion  and  avionics  subsystems. 
These  MAD  estimating  models  (see  Boeing  document  D1 94-1 0089-3  for  a 
discussion  of  the  development  of  these  models)  were  used  to  transform 
the  appropriate  failure  clock  values  in  the  baseline  F-15A  LCOM  simu¬ 
lation  data  base  to  new  values  based  on  the  equipment,  operational, 
and  environmental  parameters  contained  in  the  models.  The  MAD 
estimating  models  were  used  to  compute  new  F-clock  values  for  the 
F-15A/Bitburg  LCOM  simulation  as  follows: 

(1)  Values  for  the  various  equipment,  operational,  and 
environmental  parameters  included  in  the  MAD  estimating  equations  were 
obtained  from  the  F-15A/Bitburg  data  base  which  was  obtained  during  the 
course  of  the  study.  These  values  comprised  the  "operating  point"  for 
the  model;  i.e.,  F-15A/Bitburg  Air  Base  equipment,  operational,  and 
environmental  characteristics. 

(2)  The  linear  multiple  regression  equations  comprising  the 
MAD  estimating  models  were  then  evaluated  to  obtain  an  estimate  for 
maintenance  action  demand  per  aircraft  per  year  for  each  aircraft 
subsystem  analyzed  (initially  propulsion  and  eleven  avionic  subsystems). 
These  MAD  per  aircraft  per  year  estimates  were  then  transformed  into 
mean-sorties-to-failure  estimates  for  each  subsystem. 

(3)  The  mean-sorties-to-failure  values  from  (2)  were  then 
used  to  change  the  appropriate  F-clock  values  in  the  main  simulation 
model  by  means  of  "change  cards"  in  the  run  control  deck. 

An  initial  series  of  LCOM  simulation  experiments  were  per¬ 
formed  to  evaluate  the  Phase  I  maintenance  metrics  and  weightings. 

The  objective  of  these  experiments  was  to  determine  how  well  the 
generalized  MAD  estimating  models,  which  were  derived  from  an  Air 
Force-wide  population  of  aircraft  and  bases,  could  duplicate  actual 
historical  MAD  data  from  a  specific  aircraft  (the  F-15A)  and  a  specific 
base  (Bitburg  Air  Base,  Germany).  This  determination  is  a  measure  of 
the  confidence  that  can  be  placed  in  the  new  metrics  and  weightings 
when  used  in  a  new  situation  for  an  emerging  weapon  system.  The  deter¬ 
mination  was  made  by  exercising  the  F-15A/Bitburg  LCOM  simulation  with 
the  new  F-clock  values  singly  and  in  combination.  The  results  of 
these  simulations  were  then  compared  to  baseline  model  runs  to  determine 
how  well  the  F-clock  values  based  on  estimated  data  could  duplicate 
simulation  results  from  F-clock  values  based  on  actual  historical  data. 
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It  was  found  that  the  new  metrics  and  weightings  developed 
for  the  eleven  avionic  systems  were  able  to  duplicate  actual  historical 
results  within  plus  or  minus  10  percent.  These  estimating  models  can 
therefore  be  used  for  predicting  maintenance  action  demand  in  unknown 
situations  with  a  high  degree  of  confidence. 

The  MAD  estimating  model  for  the  propulsion  system  yielded 
large  deviations  in  simulation  results  compared  to  historical  MAD 
values.  Therefore  this  model  requires  modification  and/or  refinement 
before  it  can  be  used  with  confidence. 

The  validation  experiment  approach  outlined  above  was  next 
applied  to  the  ASD  KC-135A  LCOM  simulation  for  three  different  bases. 

These  bases  were  Loring  AFB,  Maine,  an  operational  SAC  base  in  a  cold, 
damp  climate;  Castle  AFB,  California,  a  SAC  training  base  in  a  hot, 
dry  climate;  and  Seymour-Johnson  AFB,  North  Carolina,  an  operational 
SAC  base  in  a  warm,  damp  climate.  The  three  bases  were  chosen  to 
demonstrate  the  ability  of  the  new  metrics  to  reflect  the  differences 
in  operations  and  environments  at  the  bases. 

Series  of  simulation  experiments  were  performed  to  evaluate 
the  generalized  metrics  models  developed  for  the  30  corranon  aircraft 
subsystems  selected  for  Phases  I  and  II  of  the  study.  The  simulations 
of  the  three  bases  were  first  run  using  the  metrics  currently  used  by 
ASD.  Then  the  new  metrics  model  values  were  substituted  and  the  simu¬ 
lation  runs  repeated.  Finally,  metrics  based  on  actual  1977  operational 
and  failure  data  from  the  three  bases  were  inserted  and  the  series  of 
simulations  repeated  once  more  to  form  an  LCOM  output  baseline  based 
on  base-specific,  historic  inputs. 

The  results  of  the  ASD  standard  runs  and  the  metrics  model 
runs  were  compared  against  the  results  of  the  baseline  runs.  These 
difference  analyses  determined  and  demonstrated  the  capability  of  the 
generalized  maintenance  metrics  model  inputs  relative  to  the  ASD  standard 
inputs  for  producing  similar  LCOM  outputs  to  actual  base-specific 
inputs.  The  analyses  findings  are  summarized  as  follows: 

Average  percent  difference  of  the  25 
selected  critical  output  variables  from 
the  1977  baseline  simulated  values  was  -  - 

Using  ASD  Std  Using  Maint  Metrics 

Failure  Clocks  Derived  Failure  Clocks 

Loring  AFB:  -  2.39%  -  2.85% 

Seymour-Johnson  AFB:  -  8.26%  -  8.93% 

Castle  AFB:  +1.02%  -2.79% 
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These  findings  demonstrate  that  the  generalized  metrics  models  are 
quite  acceptable  for  synthesizing  F-clock  inputs  for  the  ASD  KC-135A 
LCOM. 


Selected  output  variables  from  the  baseline  simulation  series 
of  each  base  were  then  compared  to  actual  1977  0  and  M  histories  from 
G033B  and  D056E  data  for  these  bases.  This  tested  the  overall 
fidelity  of  the  ASD  KC-135A  LCOM  simulation  used  in  the  experiments 
(this  simulation  used  a  standard  ASD  resource  data  base  instead  of 
resources  tailored  to  fit  the  conditions  at  each  specific  base).  The 
findings  of  these  comparative  analyses  were  as  follows: 

Seven  critical  0  and  M  performance  parameters  were  selected 
for  comparison,  i.e.  -  - 

o  Flying  Hours  Per  Aircraft  Per  Year 
o  Sorties  Per  Aircraft  Per  Year 
o  Average  Operational  Ready  Rate 
o  Average  Not-Operationally-Ready-Maintenance  Rate 
o  Average  Not-Operationally-Ready-Supply  Rate 
o  Total  Maintenance  Manhours  Per  Aircraft  Per  Year 
o  Average  Maintenance  Manhours  Per  Flying  Hour 

The  average  percent  deviation  of  these  parameters  as  simulated  by  the 
baseline  series  runs  of  the  KC-135A  LCOM  were  as  follows: 

Loring  AFB:  -  7.45%  average  deviation 

Seymour-Johnson  AFB:  -  9.57% 

Castle  AFB:  -  14.08% 

This  fidelity  was  considered  completely  acceptable  in  light  of  the 
standard,  non-specific  configuation  of  the  KC-135A  LCOM  simulation 
resource  base. 
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I  -  INTRODUCTION 


I.  PURPOSE  AND  SCOPE 

This  report  is  the  fourth  of  five  reports  to  be  completed  under 
the  Maintenance  Metrics  study.  It  describes  the  work  accomplished  during 
Phases  I  and  II  for  Task  VIII  as  displayed  in  Figure  1  and  enumerated 
below.  Tasks  I  through  VII  were  completed  previously  and  documented  in 
the  first  three  reports  in  this  series,  D1 9c  -.10089-1  ,  D1 94-1 0089-2  and 
D1 94-1 0089-3. 

The  significant  results  obtained  in  this  task  provide  the 
initial  validation  data  for  the  Maintenance  Metrics  models  developed  in 
previous  tasks  and  also  provide  source  data  for  related  future  research. 

The  following  is  a  brief  overview  of  the  eight  tasks  developed 
for  this  study  as  shown  in  Figure  1. 


TASK  I  Identify,  Obtain,  and  Review  Related  Publications 

-  review  related  studies  and  research  dealing 
with  maintenance  rates  and  causes. 

TASK  II  Select  Equipment 

-  develop  matrices  of  equipment  by  aircraft  type 
in  order  to  select  specific  hardware  for 
avionics  and  engines  subsystems. 

TASK  III  Identify  Parameters 

-  identify  maintenance,  hardware,  operational, 
environmental ,  and  aircraft  general  parameters 
which  would  have  an  impact  on  maintenance  for 
the  subject  subsystems. 

TASK  IV  Identify  and  Integrate  Data  Sources 

-  identify,  assemble,  correlate,  and  integrate 
the  data  base  on  the  equipment  selected  in 
Task  II  for  the  related  parameters  being 
considered  in  Task  III. 

TASK  V  Analyzing  and  Prioritizing  Parameters 

-  prioritize  the  collected  data  to  define  and 
test  relationships  between  the  study  para¬ 
meters  and  maintenance  demand  rates. 

TASK  VI  Maintenance  Metrics  Development 

-  develop  metrics  quantifying  maintenance  demand 
rates  which  are  computable  with  LOOM  models. 
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2. 


TASK  VII 

TASK  VIII 


Maintenance  Weightings  Development 

-  develop  weightings,  quantifying  identified 
impacts  upon  maintenance  demand  rates. 

Analysis  and  Modification 

-  analyze  LCQM  model  outputs  with  current  and 
the  newly  developed  metrics  and  weightings. 


BACKGROUND 


To  date,  the  manpower  and  other  resource  requirements 
essential  to  the  Operations  and  Support  ( O&S )  of  a  weapon  system  have 
been  determined  using  the  traditional  "flying  hours"  and  "sortie  rate" 
measures.  The  deficiencies  of  these  traditional  measures  are  well 
known  and  such  measures  frequently  are  found  to  be  totally  irrelevant 
(e.g.,  maintenance  on  a  gun  subsystem  is  generated  by  factors  like 
the  number  of  rounds  fired,  and  is  not  affected  by  the  number  of  flying 
hours  or  sorties).  These  traditional  measures  are  also  insensitive  to 
variations  in  operations  and  environmental  conditions  (for  example, 
many  avionics  equipments  may  operate  or  are  cycled  on  the  ground  greatly 
in  excess  of  related  flying  hours  or  number  of  sorties).  The  present 
difficulties  then  lie  in  the  fact  that  the  currently  used  metrics  do  not 
consider  the  inherent  differences  between  the  individual  subsystems  of  a 
weapon  system  and  are  relatively  insensitive  to  operational  and  environ¬ 
mental  conditions. 

The  objective  of  this  portion  of  this  research  study  is  to 
perform  initial  validation  experiments  utilizing  the  maintenance  metrics 
models  derived  during  this  study  to  generate  maintenance  demand  inputs 
for  the  Air  Force  Method  (Logistics  Composite  Model  (LCOM))  for  deter¬ 
mining  manpower  and  other  resource  requirements  for  operational  and 
developing  weapon  systems.  This  simulation  technology  has  been  docu¬ 
mented  in  References  1  through  9. 

3.  SUMMARY 


The  approach  taken  for  the  validation  of  the  maintenance 
metrics  developed  during  the  preceding  study  tasks  was  to  exercise  the 
newly  developed  metrics  in  known  historical  situation  simulations  and 
subsequently  evaluate  the  success  of  these  new  metrics  in  producing 
similar  simulation  results  as  the  actual  historical  data.  The  ability 
of  the  new  maintenance  metrics  to  duplicate  the  results  of  actual 
historical  data  is  a  measure  of  the  worth  of  these  metrics  in  predicting 
maintenance  resource  demands  for  emerging  weapon  systems  under  new 
operational  and  environmental  conditions. 

Initial  validation  experiments  were  performed  using  the 
ASD/McDonnell  Douglas  LCOM  simulation  of  the  F-15A  aircraft  at  Bitburg 
Air  Base  as  the  baseline  model.  This  model  was  executed  with  the 
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standard  failure  clock  values  which  were  derived  from  the  historical 
data  base  on  F-15A/Bitburg.  Then  a  series  of  experimental  simulation 
runs  were  executed  using  the  maintenance  metrics  and  weightings 
developed  during  this  study  to  set  the  model's  failure  clocks.  These 
experimental  runs  were  designed  to  demonstrate  the  effects  of  the  new 
metrics  singly  and  in  combination.  The  results  of  the  experimental 
simulations  were  then  compared  with  the  standard  simulations  in  order 
to  evaluate  the  worth  of  the  newly  developed  maintenance  metrics  for 
the  estimation  of  aircraft  systems  maintenance  resource  demands. 

In  the  initial  series  of  experimental  model  runs,  maintenance 
metrics  for  the  aircraft  propulsion  system  and  eleven  avionic  systems 
were  exercised.  The  results  of  this  initial  series  indicated  that  the 
avionics  metrics  were  acceptable  for  use  in  predicting  new  situations 
with  only  approximately  10%  deviation  from  the  simulation  results  given 
by  the  actual  historical  data.  The  propulsion  system  metric  indicated 
a  need  for  further  investigation  and  possible  refinement,  however,  since 
its  introduction  into  the  baseline  simulation  model  caused  wide  variations 
from  the  actual  historical  propulsion  data. 

A  more  extensive  series  of  validation  experiments  was  then 
performed  which  exercised  the  developed  metrics  for  all  thirty  aircraft 
subsystems  investigated.  A  standard  LCOM  simulation  of  the  KC-135A 
aircraft  was  used  to  simulate  three  different  bases  with  varying  environ¬ 
ments  and  operational  modes,  i.e.;  Loring  AFB,  Maine,  a  two  squadron 
operational  base;  Seymour-Johnson  AFB,  North  Carolina,  a  single  squadron 
operational  base;  and  Castle  AFB,  California,  a  two  squadron  training 
base.  These  squadrons  were  first  simulated  using  the  ASD  developed 
standard  metrics  with  base-specific  flying  programs.  Then  the  simula¬ 
tions  were  repeated  using  the  newly  developed  maintenance  metrics  from 
this  study.  Finally,  the  simulations  were  repeated  again  using  metrics 
based  on  the  historic  1977  sortie  and  failure  rates  from  the  bases  in 
question  to  form  base-specific  baseline  simulations.  Output  operational 
and  maintenance  parameters  from  the  standard  and  "new"  metrics  simulations 
were  then  compared  to  the  baselines  to  check  the  success  of  these 
metrics  in  simulating  base-specific  situations.  The  outputs  of  the 
baseline  simulations  were  in  turn  compared  to  actual  1977  O&M  histories 
at  the  subject  bases  as  extracted  from  the  Air  Force  G033B  and  D056E 
data  systems.  These  comparisons  measured  the  overall  fidelity  of 
the  ASD  KC-135A  LCOM  in  reproducing  actual  base  conditions. 

The  results  of  these  comparative  analyses  indicated  that 
the  newly  developed  maintenance  metrics  were  approximately  equal  in 
accuracy  to  the  ASD  developed  standard  KC-135A  metrics  as  measured 
against  base-specific  baseline  metrics.  Both  types  produced  simulated 
outputs  that  were  generally  within  3%  of  the  baseline  outputs  for 
Loring  and  Castle  AFB's,  and  within  9%  for  Seymour-Johnson  AFB.  The 
advantage  of  using  the  new  metrics  over  standard  methods  is  apparent  in 
new  situations  where  standard  metrics  do  not  exist.  The  standard 
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metrics  are  synthesized  from  combining  a  great  deal  of  field  failure 
data  and  take  a  significant  time  to  develop,  whereas  the  new  metrics 
are  simply  the  result  of  inserting  a  small  amount  of  parametric  data 
into  the  failure  predictor  equations  and  computing  the  predicted 
equipment  failure  rates.  This  can  usually  be  done  quickly  and  easily 
for  any  aircraft/base  situation  given  the  general  maintenance  metrics 
data  base  and  access  to  G033B  and  D056E  data. 

The  overall  fidelity  of  the  KC-135A  LCOM  as  compared  to 
actual  1977  field  data  indicated  acceptable  levels  of  under  10% 
deviation  for  Loring  and  Seymour- Johnson  AFB's,  and  under  15%  deviation 
for  Castle  AFB. 
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ANALYSIS  AND  RESULTS  OF  METRICS 
AND  WEIGHTINGS  -  TASKim 


1 .  INTRODUCTION 


Task  VIII  of  the  study  was  the  planning,  execution,  and 
analysis  of  validation  experiments  for  the  new  maintenance  metrics  and 
weightings  developed  during  the  preceding  study  tasks.  These  experi¬ 
ments  were  performed  on  operative  LCOM  simulations  of  operational  air¬ 
craft  systems.  The  validation  experiments  were  intended  to  demonstrate 
the  validity  of  the  new  metrics  and  to  set  an  approximate  confidence 
level  for  their  use. 

The  task  results  reported  herein  cover  the  validation  efforts 
for  both  Phase  I  metrics  (propulsion  and  avionics)  and  Phase  II  metrics 
(other  aircraft  systems). 

The  subtasks  accomplished  for  the  preparation  and  execution 
of  these  validation  experiments  are  as  shown  in  Figure  2  and  discussed 
in  the  following  paragraphs.  The  task  sequence  implied  by  the  flow 
shown  in  Figure  2  is  approximate.  Portions  of  several  of  subtasks  were 
actually  accomplished  in  parallel.  Figure  3  depicts  the  general 
validation  experimental  procedure. 

2.  SELECTION  OF  BASELINE  LCOM  INPUT  MODEL  -  SUBTASK  8.1 


The  first  step  in  the  process  of  analyzing  the  results  of 
metrics  and  weightings  development  effort  of  the  preceding  study  tasks 
was  the  selection  of  operative  LCOM  simulations  in  which  to  test  the 
newly  developed  metrics.  Existing  Air  Force  LCOM  simulations  were 
investigated  and  the  ASD/McDonnell  Douglas  model  of  the  F-15A  aircraft 
at  Bitburg  Air  Base  selected  for  the  initial  series  of  metrics  validation 
experiments.  The  model  selected  for  subsequent  series  of  experiments 
was  the  standard  ASD  model  of  the  KC-135A  aircraft. 

Input  models  and  flying  programs  for  the  selected  models  were 
implemented  on  the  ITEL  computer  system  in  the  ASD  Computer  Center  at 
Wright-Patterson  Air  Force  Base,  Dayton,  Ohio.  The  model  data  were 
based  on  1977  experience  data  the  same  as  the  present  study. 

3.  BASELINE  MODEL  SIMULATION  RUNS  USING  CURRENT  METRICS 

AND  WEIGHTINGS  -  SUBTASK  8.2 

After  implementation  of  the  baseline  models  on  the  ASD  comput¬ 
ing  system,  simulation  runs  were  executed  using  the  failure  clock  values 
and  maintenance  task  selection  probability  distributions  currently 
operational  in  the  input  data  bases  for  the  models.  These  runs  served 
to  calibrate  the  natural  variability  of  the  baseline  simulations  and  to 
establish  a  basis  for  comparison  of  the  results  of  the  later  validation 
experiments  which  utilized  the  newly  developed  F-clock  metrics. 
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FIGURE  2  TASK  VIII  PROCESS  FLOW 


FIGURE  3  MAINTENANCE  METRICS  VALIDATION  EXPERIMENT  PROCEDURE 


PROCEDURE  FOR  TRANSFORMING  BASELINE  LCOM  FAILURE  CLOCKS  - 
SUBTASk  8.3 


The  next  step  of  the  validation  process  was  to  implement  a 
procedure  for  transforming  the  baseline  failure  clock  values  in  the 
test  models  to  values  computed  from  the  F-clock  estimation  equations 
developed  in  preceding  study  tasks  VI  and  VII.  The  procedure  developed 
utilized  the  "change-card"  capability  of  the  LCOM  control  software  so  as 
to  facilitate  ease  of  testing  various  combinations  of  modified  clock 
values  without  disturbance  to  the  basic  baseline  Input  Data  Model.  This 
procedure  is  as  follows: 

PROCEDURE  FOR  TRANSFORMING  PRESENT  LCOM 

Failure  clock  vaLUFS  TO  CONFORM  with  ' 

Maintenance  metrics  model  Estimates 


(1)  Determine  actual  historical  time  period  used  to  derive  present 
LCOM  values. 

(2)  Determine  actual  maintenance  action  demand  (AMAD)  of  item  of  interest 
during  that  time  period. 

(3)  Determine  appropriate  "operating  point"1  values  for  item's  Metrics 
Model  regression  variables.  These  values  may  either  be  derived 
from  historic  design  and  scenario  data  or  from  new  simulated  design 
and  scenario  data  as  appropriate  depending  on  the  nature  of  the 
simulation  experiments  to  be  performed. 

(4)  Compute  estimated  maintenance  action  demand  (EMAD)  for  the  same 
historic  time  period  using  Maintenance  Metrics  Regression  Model. 

(5)  Compute  ratio  of  EMAD  to  AMAD. 


NOTE: 

1  -  Operating  point  is  defined  here  as  the  system  of  design,  operational 
support,  and  environmental  conditions  applicable  to  the  item-of- 
interest.  This  may  be  some  actual  historic  operating  point  featuring 
retrospective  data,  a  predicted  operating  point  featuring  prospec¬ 
tive  estimates,  or  it  may  be  a  mixture  of  the  two. 
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(6)  Multiply  present  clock  values  (or  decrement  value  if  appropriate) 
by  the  EMAD/AMAD  ratio  to  transform  clock  value  to  the  Maintenance 
Metric  based  estimate. ^ 

(7)  If  new  clock  value  is  to  be  substituted  into  an  existing  LCOM 
input  model  and  it  is  desired  not  to  disturb  the  existing  input 
data  base,  add  a  clock  change  card  to  the  LCOM  simulation  control 
deck  designating  the  appropriate  clock  number  and  new  clock  value. 

A  typical  example  of  the  application  of  this  procedure  to  the 
F-15A/Bitburg  baseline  LCOM  is  as  follows: 

EXAMPLE  OF  FAILURE  CLOCK  TRANSFORMATION  PROCEDURE: 

Assume  that  there  exists  a  failure  clock  for  the  F-15A  Flight 
Indicators  Subsystem  (WUC-51A)  which  is  based  on  1977  maintenance  demand 
and  sortie  data  from  Bitburg  Air  Base. 

Step  1  Derivation  time  period  =  1977 

Step  2  Actual  maint.  action  demand  (AMAD)  for  WUC-51A: 

(LCOM  definition  AMAD  per  system  per  year) 

(Source:  AFM  66-1  (D056E)  data  for  1977) 

LCOM  Task  Code  R  =  46  actions/32  systems  =  1.43750 

LCOM  Task  Code  M  =  20  actions/32  systems  =  0.62500 

LCOM  Task  Code  H  =  11  actions/32  systems  =  0.34375 

Total  1977  AMAD  (LCOM  Definition)  2.40625 

Step  3  1977  values  for  significant  F-15A  (WUC-51A)  Maintenance  Metrics 

Regression  Model  variables  (Bitburg  data): 


Equipment  Variables: 

A03,  Equipment  Weight . 0.72  lbs. 

Operations  Variables: 

013,  Minimum  Landing  Distance . 3750.00  feet 


017,  Operations  Flying  Hours  per  Aircraft  .  223.53  hrs./yr. 


NOTE: 

2  -  The  Maintenance  Metrics  Models  are  of  greatest  value  when  performing 
prospective  simulation  and  analyses  on  new  systems  and/or  new 
scenarios.  Under  these  conditions  it  is  postulated  that  they  will 
provide  better  results  than  simplistic  projections  of  historic 
failures  per  sortie  or  per  flying  hour.  If,  however,  an  exact 
historical  scenario  is  being  simulated  (a  retrospective  analysis  of 
what  actually  happened),  the  historical  data  should  provide  better 
results  than  the  "fitted"  Maintenance  Metrics  estimates. 
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Environmental  Variables: 

E03,  Runway  Direction  .  240.00  compass 

degree 

E19,  Maximum  Crosswinds  20-29  mph  .  106.00  days/yr. 


Step  4  Estimated  maint.  action  demand  (EMAD)  for  WUC-51A: 

(F-15A  Bitburg  Situation,  1977) 

WUC-51A  Maint.  Metrics  Regress  Model: 

(Derived  from  data  for  WUCs  51  AD,  51 AH,  and  51 AK) 

EMAD  =  4 .65791 +(0 .3981 3) (0.72 )+(0. 00036) (3750.0)+. . . 

... +(0. 001 59) (223. 53)- (0.00361) (240. 0)+(0. 04497) (106.0) 

EMAD  (for  51AD,  51AH,  51 AK)  =  1.23458  actions  per  year 

AMAD  (for  51  AD ,  51AH,  51 AK )  =  0.88  actions  per  yr  (from  66-1  data) 

Ratio  of  total  51 A  AMAD  to  partial  AMAD  above: 

2.40625/0.88  =2.73 

Total  51 A  EMAD  =  (2 .73) (1 .23458)  =  3.376 

Step  5  Ratio  of  total  WUC-51A  EMAD  to  AMAD 
3.376/2.406=1.403 

Step  6  Calculation  and  transformation  of  baseline  failure  clock  value: 

Assume  that  the  baseline  WUC-51  failure  clock  value  is  based  on 
sorties  per  failure  for  the  year  1977  with  no  allowance  for 
peak  sortie  rate  or  peak  failure  rate  periods. 

Then— Sorties  per  Failure  =  Total  Sorties  per  Acft/Total  AMAD 

per  unit 
=  174.53/2.406 
=  72.54 

Set  baseline  F-clock  at  73  sorties  to  failure 

Transformed  F-clock  value  =  (AMAD/EMAD)  (Baseline  Clock  Value) 

=  (1 .403) (72 .54) 

=  101.77 

Set  new  F-clock  value  at  102  sorties  to  failure  by  adding  a 
clock  change  card  to  the  LC0M  control  deck  designating  the 
appropriate  clock  number  and  the  new  clock  value. 

Initially,  the  procedure  was  applied  to  the  propulsion  and 
eleven  of  the  avionics  failure  clocks  of  the  F-15A/Bitburg  baseline 
model.  Figure  4  displays  the  key  to  the  F-clock  transformation  work¬ 
sheet  used  to  record  the  calculations  involved.  The  completed  worksheet 
of  the  F-clock  transformations  for  the  initial  F-15A/Bitburg  validation 
experiments  is  included  as  Appendix  A.  The  resulting  F-clock  values 
and  their  implications  for  the  baseline  F-15A/Bitburg  LC0M  are  sunmarized 
in  Table  1.  Baseline  values  for  the  subject  F-clocks  had  been  calculated 
from  1977  Bitburg  data  prior  to  the  model's  use  in  the  metrics  study. 
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APPLICATION  OF  METRIC  MODELS  TO  F-15A  (BITBURG) 
LCOM  FAILURE  CLOCKS  (PHASE  I  EQUIPMENTS) 


Therefore,  that  portion  of  step  6  was  not  necessary  for  the  initial 
experiments.  The  values  for  the  regression  variables  were  obtained  from 
the  F-15A/Bitburg  entries  in  the  Maintenance  Metrics  study  data  base. 
These  transformed  F-clocks  were  used  according  to  the  validation 
experiment  plan  of  paragraph  II. 5. 

The  F-clock  transformation  procedure  was  then  applied  to  all 
30  aircraft  subsystems  studied  for  the  LOOM  simulations  of  the  three 
selected  KC-135A  bases.  The  simulation  model  used  for  these  experiments 
contained  generic  ASD  standard  F-clock  values  derived  from  a  composite 
of  five  representative  KC-135A  bases;  i.e.,  Altus,  Blytheville,  Grand 
Forks,  Griffiss,  and  K.  I.  Sawyer  (See  reference  10).  Therefore  it 
was  necessary  to  calculate  sets  of  base-specific  baseline  F-clock 
values  for  the  three  study  bases;  Loring,  Seymour-Johnson ,  and  Castle. 
Sortie  and  failure  data  from  the  year  1977  were  used  for  this  purpose. 
The  D056E,  G033B,  and  KC-135A  source  data  used  for  calculation  of  the 
baseline  failure  clocks  and  also  for  use  in  the  F-clock  transformation 
regression  equations  is  included  in  this  document  as  Appendix  B. 

Appendix  C  contains  the  baseline  F-clock  calculation  worksheets.  These 
baseline  F-clock  values  were  then  imposed  on  the  existing  generic  ASD 
KC-135A  model  via  appropriate  clock  change  cards  for  the  base-specific 
baseline  simulation  runs. 

The  thirty  study  equipment  failure  clocks  were  then  trans¬ 
formed  to  the  maintenance  metrics  values  for  the  metrics  validation 
experiments.  Appendix  D  contains  the  completed  worksheets  for  these 
transformations  for  each  of  the  three  study  bases.  The  values  for  the 
regression  variables  in  these  worksheets  were  obtained  from  the  subject 
base  entries  in  the  1977  G033B,  D056E,  and  Air  Weather  Service  data 
bases  for  maintenance  demand,  operations,  and  environmental  variables. 
KC-135A  equipment  design  characteristic  data  were  obtained  from  the 
Maintenance  Metrics  study  data  base.  Table  2  contains  a  summary  of  the 
ASD  standard,  baseline,  and  metrics  derived  F-clock  values  for  each  of 
the  study  bases.  The  validation  experiment  plan  based  on  these 
transformed  F-clock  values  is  given  in  paragraph  1 1. 7. 

5.  NEW  METRICS  AND  WEIGHTINGS  INITIAL  VALIDATION  EXPERIMENTS  - 

SUBfASKS  8.4 


An  initial  series  of  LCOM  simulation  experiments  were  per¬ 
formed  to  evaluate  the  F-clock  estimation  equations  developed  during 
study  tasks  VI  and  VII  (refer  to  Boeing  document  D1 94- 1 0089-3 )  for  the 
Phase  I  study  equipments  (propulsion  and  avionics).  These  equations 
appear  on  the  F-clock  transformation  worksheet  in  Appendix  A. 

The  objective  of  these  experiments  was  to  determine  how  well 
the  generalized  F-clock  estimating  models,  which  were  derived  from  an 
Air  Force-wide  population  of  aircraft  and  bases,  could  duplicate  simula¬ 
tion  results  based  on  actual  historical  failures  per  sortie  data  from 
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TABLE  2  SUMMARY  OF  F-CLOCK  VALUES  TRANSFORMED 
FOR  KC-135A  LOOM  METRICS  VALIDATION  EXPERIMENTS 


SYSTEM 

F-CLOCK 

NUMBER 

I.D. 

F-CLOCKS 

IN  ASS 

KC-13SA  MODEL 

F-CLOCKS 

CORING 

3ASELINE 

F-CLOCKS 

LCRING 

METRICS 

F-CLOCKS 

SEYMOUR-u 

3ASELINE 

F-CLOCKS 

SEYMOUR-J 

'•ETRICS 

F-CLOCKS 

CASTLE 

3ASE11NE 

F-CLOCKS 

CASTLE 

METRICS 

Propulsion 

FA23AS 

25.0 

38.5 

37.7 

29.0 

51.7 

23.4 

47.4 

FA23AO 

567.0 

789.5 

773.7 

'32.0 

1 395 . 3 

608.6 

1016.4 

FA23SS 

9.0 

29.3 

29.2 

6.3 

’1.2 

3.1 

13.5 

FA23CS 

103.0 

17.5 

17.2 

11.7 

20.9 

13.6 

31  .1 

FA230S 

174.0 

42.7 

41.3 

50.2 

107.4 

52.5 

37.7 

FA23ES 

10.4 

32.2 

31  .6 

10.4 

’8.5 

9.2 

15.4 

FA23HS 

15.0 

10.4 

10.2 

4.6 

3.2 

7.9 

13.2 

FA23JS 

9.0 

5.1 

5.0 

2.7 

4.3 

4.7 

7.3 

FA23J0 

1134.0 

789.5 

773.7 

391.0 

697.6 

608.6 

1016.4 

FA23KS 

4.0 

6.1 

6.0 

3.7 

o.2 

6.  ‘ 

'0.2 

FA23LS 

19.0 

7.4 

7.3 

7.2 

i2. a 

?  .  4 

15.7 

FA23MS 

7.0 

6.4 

5.3 

5.2 

9.3 

_  7 

5.2 

FA23NS 

16.0 

11.0 

10.3 

35.5 

53.3 

11.3 

.3.9 

FA230S 

39.0 

225.6 

221.1 

10.6 

13.9 

50.7 

34.7 

FA23PS 

5.0 

4.9 

4.3 

3.9 

7.3 

5.2 

3.7 

FA23RS 

13.0 

8.3 

3.1 

5.7 

10.2 

VO 

11.7 

FA23RO 

73.3 

49.3 

48.3 

34.0 

60.  7 

42.3 

•0.6 

Fit.  Indie. 

FAS11S 

7.3 

11.0 

22.0 

7.6 

•t  2 . 2 

7.5 

*  7.5 

Air  Data 

FAS IBS 

19.0 

20.5 

6.3 

12.6 

14.5 

13.3 

53.7 

Horiz.  Situa 

FAST AS 

4.5 

7.5 

4.5 

5.3 

25.7 

4.’ 

6.5 

Autopilot 

FA521S 

18.0 

27.2 

20.1 

41.2 

19.6 

26.2 

43.0 

FA5210 

5.3 

9.1 

5.7 

13.5 

6.4 

3.3 

14.4 

UHF  Comm. 

FA63RS 

37.0 

4.4 

*41 .3 

7.7 

*51.3 

7.3 

•12.5 

IFF  Set 

FA65BS 

10.6 

17.2 

30.4 

11.3 

23.6 

15.3 

36.3 

Inst.  Lndg. 

FA71BS 

13.6 

21.6 

9.7 

41.2 

25.7 

27.4 

14.5 

Tacan 

FA71CS 

5.7 

7.4 

16.6 

5.3 

39.5 

10.3 

3.3 

Radar 

FA723S 

1.3 

2.3 

2.5 

2.2 

1.5 

2.9 

9.7 

Fuselage 

FA11 IS 

450.0 

4.2 

0.9 

6.7 

9.8 

7.3 

3.1 

Wings 

FA11A0 

18.0 

21.1 

15.6 

17.0 

49.5 

18.8 

16.2 

FA11JO 

7.0 

7.7 

5.7 

3.1 

9.0 

3.9 

7.7 

FA11K0 

7.3 

8.0 

5.9 

2.8 

8.2 

11.9 

10.2 

FA116S 

99.0 

131.6 

97.4 

130.3 

379.7 

86.9 

74.7 

FA1160 

44.0 

65.3 

48.7 

71.1 

207.2 

43.5 

37.4 

FA117S 

103.0 

121.5 

89.9 

156.4 

455.8 

144.9 

124.6 

FA1 1 70 

37.0 

41.6 

30.3 

55.9 

162.9 

48.3 

41.5 

Cockpit 

FA12AS 

67.0 

83.1 

•  75.6 

71.1 

201.4 

138.3 

*120.0 

Furnishings 

FA12AO 

142.0 

157.9 

143.7 

156.4 

443.1 

276.6 

240.0 

Lndg.  Gear 

FA1 3AO 

3.5 

3.0 

1.9 

3.3 

3.0 

2.5 

4.3 

Brakes 

FA13CS 

3.0 

4.7 

*22.4 

10.7 

16.8 

5.4 

36.9 

Stabilator 

FA11SO 

27.0 

23.6 

17.0 

60.2 

14.0 

53.4 

19.2 

Rudder 

FA1 480 

69.0 

8.9 

*  9.6 

24.4 

*  3.7 

14.2 

♦  5.3 

Flaps 

FA14E0 

11.0 

3.0 

*  2.2 

4.8 

*  5.0 

6.0 

*15.7 

Envl ron . 

FA412S 

18.6 

38.5 

*55.0 

32.6 

*186.0 

31.1 

*44.4 

Control 

FA4120 

26.0 

56.4 

30.7 

48.9 

279.2 

46.1 

65.9 

Elect.  Pwr. 

FA421S 

38.0 

4.4 

12.3 

2.8 

10.3 

4.2 

12.9 

Hydr.  Pwr. 

FA451S 

3.0 

3.2 

*  4.5 

5.6 

*  6.0 

3.7 

*19.0 

Internal 

FA451S 

12.0 

12.0 

*49.1 

31.3 

55.2 

14.6 

*833.1 

Fuel 

FA4620 

13.0 

10.7 

43.3 

39.1 

53.9 

12.7 

724.6 

FA4630 

23.0 

22.6 

92.5 

71.1 

125.3 

17.6 

1004.2 

to*  Syst. 

FA471S 

10.0 

13.4 

24.9 

11.3 

*45.8 

14.4 

♦50.9 

FI  re  Select. 

FA494S 

16.3 

12.7 

*187.6 

11.3 

*460.9 

7.3 

*757.4 

•from  other  than  composite  mod*!  (sh  Appendix  0) 
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a  specific  aircraft  (F-15A),  a  specific  base  (Bitburg),  and  a  specific 
time  period  (1977).  This  determination  is  a  measure  of  the  confidence 
that  can  be  placed  in  the  estimating  equations  when  used  in  a  new 
situation  or  for  an  emerging  weapon  system.  The  determination  was  made 
by  exercising  the  F-15A/Bitburg  ICOM  simulation  with  the  new  F-clock 
values  singly  and  in  combination.  The  results  of  these  simulations  were 
then  compared  to  baseline  model  runs  as  discussed  in  paragraph  II. 6. 

This  initial  series  of  experimental  simulation  runs  to  validate  the 
Phase  I  F-clock  estimation  equations  is  listed  in  Table  3.  The  re¬ 
lationships  of  these  initial  validation  experiments  are  shown  in  the 
simulation  plan  shown  in  Figure  5.  The  simulation  experiments  were 
performed  at  the  ASD  Computer  Center,  Wright-Patterson  Air  Force  Base, 
Ohio,  on  the  ITEL  Computer  System. 

6.  DIFFERENCE  ANALYSIS  -  BASELINE  VERSUS  MODIFIED  F-15A  MODEL 

RESULTS  [NEW  METRICS)  -  SUBTASK  8TT~ 

At  the  conclusion  of  the  initial  Phase  I  validation  experi¬ 
ments,  a  difference  analysis  was  performed  which  compared  the  results  of 
the  baseline  simulation  with  the  various  experimental  runs  as  listed  in 
Table  3.  This  analysis  determined  how  well  the  F-clock  values  based  on 
estimated  data  could  duplicate  simulation  results  from  F-clock  values 
based  on  actual  historical  data.  The  analysis  compared  twenty-five 
critical  output  variables  of  the  baseline  run  against  the  same  outputs 
of  the  various  experimental  runs.  Table  4  lists  the  critical  output 
variables  monitored. 

In  the  initial  series  of  Phase  I  validation  runs,  it  was  found 
that  the  new  F-clock  estimating  equations  developed  for  the  eleven 
avionic  systems  were  able  to  duplicate  actual  historical  results  within 
approximately  plus  or  minus  10  percent.  It  is  therefore  considered  that 
these  estimators  can  be  used  for  predicting  F-clock  values  in  new  situa¬ 
tions  with  a  high  degree  of  confidence. 

The  F-clock  estimating  equation  for  the  propulsion  system 
yielded  significant  deviations  in  simulation  results  compared  to  the 
baseline  run.  Therefore,  it  was  considered  that  this  estimating 
equation  required  modification  and/or  refinement  before  it  can  be  used 
with  confidence. 

Table  5  summarizes  the  findings  of  the  difference  analysis 
of  the  initial  series  of  Phase  I  validation  experiments. 

7.  NEW  METRICS  AND  WEIGHTINGS  SUBSEQUENT  VALIDATION  EXPERIMENTS  - 
Subtask  8.4 


Three  subsequent  series  of  LCOM  simulation  experiments  were 
performed  to  evaluate  the  thirty  F-clock  estimation  equations  developed 
during  study  tasks  VI  and  VII  (refer  to  Boeing  document  0194-10089-3) . 
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EXPERIMENTAL  LCOM  RUNS  (F-15A,  BITBUR6) 
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INITIAL  SERIES— ASD/MCDONNELL  DOUGLAS  LCOM  SIMULATION 
OF  F-15A/BITBURG  (1977  DATA  BASE) 


SERIES  1  BASELINE  RUN  (1977  DATA,  20  HR.  FLYING  PROGRAM) 
F-CLOCKS  BASED  ON  1977  HISTORICAL  FAILURES/SORTIE. 


SERIES  1  VALIDATION  EXPERIMENTS.  F-CLOCKS  OF  NETWORKS  TESTED 

ARE  BASED  ON  METRICS  MODELS  WITH  1977  BITBURG  OPERATING  POINT. 

EXPERIMENT  1:  ALL  AVIONICS  AND  PROPULSION  NETWORKS  TESTED. 


Experiment  2:  large  difference  (metrics  to  baseline 

F-CLOCK  VALUES)  NETWORKS  TESTED. 

‘EXPERIMENT  3:  ENGINE  NETWORKS  #1  AND  #2  tested. 
‘EXPERIMENT  4:  ALL  AVIONICS  NETWORKS  TESTED. 


ENGINE  #1  NETWORK  METRIC  TESTED  SINGLY. 
FLIGHT  INDICATOR  F-CLOCK  METRIC  TESTED. 
UHF  SET  F-CLOCK  METRIC  TESTED. 
ATTITUDE-HEADING  REF.  SET  F-CLOCK 


‘EXPERIMENT  5 

‘EXPERIMENT  6 

‘EXPERIMENT  7 

‘EXPERIMENT  8 

METRIC  TESTED. 

‘EXPERIMENT  9:  INERTIAL  NAV  SET  F-CLOCK  METRIC  TESTED. 

‘EXPERIMENT  10:  AIR  DATA  SYSTEM  F-CLOCK  METRIC  TESTED. 

‘EXPERIMENT  11:  HORIZONTAL  SITUATION  INDIC.  F-CLOCK 
METRIC  TESTED. 


‘EXPERIMENT  12 

‘EXPERIMENT  13 

‘EXPERIMENT  14 
TESTED. 


AUTOPILOT  F-CLOCK  METRIC  TESTED. 
TACAN  SET  F-CLOCK  METRIC  TESTED. 
IFF  TRANSPONDER  SET  F-CLOCK  METRIC 


FIGURE  5  TASK  VIII  -  INITIAL  VALIDATION 
EXPERIMENTS  SIMULATION  PLAN 
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TABLE  4 

CRITICAL  OUTPUT  VARIABLES  MONITORED 
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22.  NUMBER  OF  ITEMS  BACKLOGGED 

23.  NUMBER  OF  UNITS  DEMANDED 

21.  PERCENT  OF  DEMANDS  NOT  SATISFIED 
25.  NUMBER  OF  ITEMS  ON  BACKORDER 


SERIES  1  DIFFERENCE  ANALYSIS  SUMMARY 
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SERIES  1  DIFFERENCE  ANALYSIS  SUMMARY  (CONTINUED) 


r 


These  experiments  were  designed  to  test  the  new  metrics  equations  within 
the  context  of  an  aircraft  type  (cargo-tanker)  and  subsystem  assemblage 
which  was  quite  different  than  the  baseline  aircraft  subsystem  configura¬ 
tion  around  which  the  equations  were  originally  developed,  i.e.,  the 
F-15A  fighter-interceptor.  Also,  the  experiments  pertained  to  Air  Force 
base  simulations  (Lori.ng,  Seymour-Johnson,  and  Castle)  which  were  not 
included  in  the  original  study  data  base.  Application  to  these  bases 
forms  a  significant  check  on  the  applicability  of  the  equations  to  new 
basing  situations  and  gives  indication  of  the  relevant  range  of  the 
derived  F-clock  estimation  models.  The  thirty  subject  equations  are 
listed  by  subsystem  in  Table  D1 ,  Appendix  D,  with  Loring  AFB  operating 
point  values  for  the  included  variables.  Table  D2  lists  the  equations 
with  Seymour-Johnson  AFB  values,  and  Table  D3  lists  the  equations  with 
Castle  AFB  values.  The  F-clock  values  for  each  base  resulting  from 
the  computations  of  Tables  D1 ,  D2,  and  D3  are  summarized  in  Table  2, 
paragraph  II. 4. 

As  in  the  initial  series  of  experiments,  the  objective  of 
these  simulations  was  to  determine  the  expected  accuracy  and  confidence 
level  to  be  placed  on  estimates  computed  from  the  new  metrics  models 
when  used  in  a  new  situation  or  for  an  emerging  weapon  system.  The 
validation  experiments  were  planned  to  exercise  the  KC-135A/Loring, 
Seymour-Johnson,  and  Castle  LCOM  simulations  with  the  new  F-clock 
values  to  test  the  sensitivity  of  the  simulation  results  to  the  metrics 
inputs.  The  results  of  these  simulations  were  then  compared  to  base¬ 
line  model  runs  and  to  actual  historical  1977  performance  data  from 
the  subject  bases  as  discussed  in  paragraph  II. 8.  The  series  of 
planned  experimental  simulation  runs  and  their  relationships  for  the 
KC-135A/Loring  LCOM  are  depicted  in  Figure  6.  Similar  simulation 
plans  for  the  KC-135A/Seymour-Johnson  LCOM  and  the  KC-135A/Castle  LCOM 
are  depicted  in  Figures  7  and  8.  Three  simulation  runs,  each  using  a 
different  clock  control  random  number  seed,  were  executed  for  each  set 
of  standard,  baseline,  and  metrics  validation  runs.  The  code  names 
of  these  runs  are  shown  on  the  simulation  plans  (Figures  6,  7,  and  8). 

The  three  runs  for  each  set  were  necessary  to  average  out  random 
deviations  in  the  simulation  outputs  and  allow  a  more  accurate  compari¬ 
son  of  results.  The  depicted  plans  are  meant  to  be  progressive 
depending  upon  the  results  obtained  from  the  initial  experiments  in  the 
series.  For  instance,  if  the  results  of  experiment  1  of  simulation 
series  2  (refer  to  Figure  6),  where  all  30  F-clocks  are  modified  and 
tested  together,  indicate  no  significant  deviations  from  the  historic 
performance  data  to  be  used  for  comparison,  further  experimentation 
would  not  be  required.  If,  however,  significant  deviation  was  detected, 
then  further  experimentation  according  to  the  plan  would  be  required 
to  identify  the  particular  F-clocks  causing  the  deviation.  As  in  the 
initial  series,  the  experiments  in  these  subsequent  series  were  per¬ 
formed  at  the  ASD  Computer  Center,  Wright-Patterson  AFB. 
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MODEL  USED — ASD  STANDARD  LOOM  SIMULATION  OF  KC-135A/LORING  AFB 
SEYMOUR  JOHNSON  AFB/ CASTLE  AFB  (1977  DATA  BASE) 


.SERIES  2  COMPARISON  RUNS 

STANDARD  RUNS-  LORING  AFB  (1977  DATA,  7,481  HR.  FLYING  PROGRAM) 
F-CLOCKS  SET  AT  STANDARD  KC-135A  VALUES.  (LSEED1 ,  LSEED2,  LSEED3) 

,  BASELINE  RUNS-  F-CLOCKS  SET  AT  L0RING  BASELINE  VALUES. 

(LSEED7 ,  LSEED8,  LSEED9) 

.'SERIES  2  VALIDATION  EXPERIMENTS.  KC-135A/LORING  AFB  1977 

OPERATING  POINT.  F-CLOCKS  OF  NETWORKS  TESTED  OBTAINED  FROM 
METRICS  MODELS. 


OPTIONAL 

FOLLOW-ON 

EXPERIMENTS 


EXPERIMENT  1:  ALL  30  STUDY  NETWORKS  TESTED. 

(LSEED4,  LSEED5,  LSEED6) 

, _ ^EXPERIMENT  2:  LARGE  DIFFERENCE  (METRICS  TO  STANDARD 

F-CLOCK  VALUES)  NETWORKS  TESTED. 


'EXPERIMENT  3:  AVIONICS  NETWORKS  TESTED. 


EXPERIMENT  4:  PROPULSION  NETWORKS  TESTED. 


EXPERIMENT  5:  AIRFRAME  NETWORKS  TESTED. 


‘EXPERIMENT  6:  FLIGHT  CONTROL  SYSTEMS  NETWORKS  TESTED. 


‘EXPERIMENT  7:  UTILITY  SYSTEMS  NETWORKS  TESTED. 


FIGURE  6  TASK  VIII  -  KC-135A/L0RING  AFB 
VALIDATION  EXPERIMENTS  SIMULATION  PUN 
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MODEL  USED— ASD  STANDARD  LOOM  SIMULATION  OF  KC-135A/L0RING  AFB 
SEYMOUR  JOHNSON  AFB/ CASTLE  AFB  (1977  DATA  BASE) 


, —  SERIES  3  COMPARISON  RUNS 

STANDARD  RUNS-  SEYMOUR  JOHNSON  AFB  (1977  DATA,  3,778  HR.  FLYING 
PROGRAM).  F-CLOSKS  SET  AT  STANDARD  KC-135A  VALUES. (SSEED1 ,  SSEED2,  SSEED3) 
,  BASELINE  RUNS-  F-CLQCKS  SET  AT  SEYMOUR-JOHNSON  BASELINE  VALUES. 

(SSEED7,  SSEED8,  SSEED9) 

, — .SERIES  3  VALIDATION  EXPERIMENTS.  KC-135A/SEYMOUR  JOHNSON  AFB 
1977  OPERATING  POINT.  F-CLOCKS  OF  NETWORKS  TESTED  OBTAINED 
FROM  METRICS  MODELS. 


.EXPERIMENT  1: 


OPTIONAL 

FOLLOW-ON 

EXPERIMENTS 


ALL  30  STUDY  NETWORKS  TESTED. 
(SSEED4,  SSEED5,  SSEED6) 


EXPERIMENT  2:  LARGE  DIFFERENCE  (METRICS  TO  STANDARD 
F-CLOCK  VALUES)  NETWORKS  TESTED. 


EXPERIMENT  3:  AVIONICS  NETWORKS  TESTED. 


‘EXPERIMENT  4:  PROPULSION  NETWORKS  TESTED. 


EXPERIMENT  5: 


EXPERIMENT  6: 


EXPERIMENT  7: 


AIRFRAME  NETWORKS  TESTED. 

FLIGHT  CONTROL  SYSTEMS  NETWORKS  TESTED. 
UTILITY  SYSTEMS  NETWORKS  TESTED. 


FIGURE  7  TASK  VIII  -  KC-135 A/SEYMOUR  JOHNSON  AFB 
VALIDATION  EXPERIMENTS  SIMULATION  PLAN 
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MODEL  USED— ASD  STANDARD  LOOM  SIMULATION  OF  KC-1 35A/L0RING  AFB 
SEYMOUR  JOHNSON  AFB/CASTLE  AFB  (1977  DATA  BASE) 


, _ SERIES  4  COMPARISON  RUNS 

STANDARD  RUNS-  CASTLE  AFB  (1977  DATA, 18,372  HR.  FLYING  PROGRAM) 
F-CLOCKS  SET  AT  STANDARD  KC-135A  VALUES .(CSEED1 ,  CSEED2,  CSEED3) 
BASELINE  RUNS-  F-CLOCKS  SET  AT  CASTLE  BASELINE  VALUES. 

(CSEED7,  CSEED8,  CSEED9) 

SERIES  4  VALIDATION  EXPERIMENTS.  KC-1 35A/ CASTLE  AFB  1977  OPERATING 
POINT.  F-CLOCKS  OF  NETWORKS  TESTED  OBTAINED  FROM  METRICS  MODELS. 


EXPERIMENT  1:  ALL  30  STUDY  NETWORKS  TESTED. 

(CSEED4,  CSEED5,  CSEED6) 

^EXPERIMENT  2:  LARGE  DIFFERENCE  (METRICS  TO  STANDARD 
F-CLOCK  VALUES)  NETWORKS  TESTED. 


OPTIONAL 

FOLLOW-ON 

EXPERIMENTS 


EXPERIMENT  3:  AVIONICS  NETWORKS  TESTED. 

'EXPERIMENT  4:  PROPULSION  NETWORKS  TESTED. 

'EXPERIMENT  5:  AIRFRAME  NETWORKS  TESTED. 

‘EXPERIMENT  6:  FLIGHT  CONTROL  SYSTEMS  NETWORKS  TESTED. 
'EXPERIMENT  7:  UTILITY  SYSTEMS  NETWORKS  TESTED. 


FIGURE  8  TASK  VIII  -  KC-135A/CASTLE  AFB 
VALIDATION  EXPERIMENTS  SIMULATION  PLAN 
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8. 


DIFFERENCE  ANALYSIS  -  BASELINE  VERSUS  MODIFIED  KC-135A 


As  the  series  2,  3,  and  4  validation  experiments  were 
performed,  difference  analyses  were  performed  which  compared  the 
results  of  the  baseline  simulations  of  the  three  subject  bases,  Loring, 
Seymour-Johnson,  and  Castle  with  the  various  experimental  runs  as 
depicted  in  Figures  6,  7,  and  8.  These  simulation  results  were  also 
compared  with  actual  historical  squadron  performance  data  from  the 
1977  time  period  simulated.  These  analyses  indicated  how  well  the 
F-clock  values  based  on  estimated  metrics  data  could  simulate  the 
actual  historic  situation  as  compared  to  the  current  standard  F-clock 
values  used  in  the  baseline  simulations.  The  analyses  compared  25 
critical  output  variables  (see  Table  4,  paragraph  II. 6)  of  the 
baseline  runs  against  the  same  outputs  of  the  various  experimental 
and  standard  runs.  Selected  operational  and  maintenance  (O&M) 
critical  output  variables  from  the  baseline  runs  were  then  compared 
against  actual  1977  values  from  the  historic  data  files  from  the 
subject  bases  in  the  G033B  and  D056E  Air  Force  data  systems. 

Figure  9  depicts  the  relationships  of  the  comparisons  made.  Tables 
6,  7,  and  8  present  the  numerical  results  of  the  difference  analyses 
depicted  by  Figure  9  for  having  Seymour-Johnson,  and  Castle  AFB's 
respectively.  Summary  findings  of  these  difference  analyses  are 
presented  in  Tables  9,  10,  and  11.  Comparisons  of  each  base's 
baseline  simulation  output  to  1977  actual  data  are  given  in  Tables 
12,  13,  and  14. 

The  general  procedure  used  in  the  comparative  analyses 
depicted  in  Figure  9  was  as  follows: 

(1)  Record  the  critical  variable  outputs  for  each  set  of 
three  random-number-seed  runs  for  standard,  metrics,  and  baseline 
F-clock  simulations.  These  data  are  recorded  in  Appendix  E. 

(2)  Average  the  outputs  for  each  set  of  three  seeds  for 
the  standard,  metrics,  and  baseline  run  sets.  These  averages  are 
the  values  used  in  the  difference  analyses  recorded  in  Tables  6,  7, 
and  8. 

(3)  Compare  difference  in  critical  outputs  between  the 
standard,  metrics,  and  baseline  runs  for  each  base.  Tables  6,  7, 
and  8  show  these  differences  for  Loring,  Seymour-Johnson,  and 
Castle  Air  Force  Bases  respectively. 

(4)  Summarize  and  interpret  the  findings  of  the  difference 
analyses  for  each  base.  These  summaries  for  Loring,  Seymour- 
Johnson,  and  Castle  AFB's  appear  as  Tables  9,  10,  and  11. 
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FIGURE  9  RELATIONSHIPS  AND  PROCEDURE  FOR  KC-135A 
METRICS  VALIDATION  COMPARATIVE  ANALYSIS 


TABU  6  DIFFERENCE  ANAETS1S  FOR  LCOH  VAllDATION  EXW.RIMLNIS,  197?  DATA  -  tORINIi  AFB,  KC-UbA 
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TABLE  7  OIF«R£NC£  ANALYSIS  fOR  ICON  VALIDATION  £X««l/ttN7S.  19;?  BATA  -  Sf YNOOR-JWINSON  ALB,  KC- USA 
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TABLE  13  COMPARISON  OF  CRITICAL  KC-I35A  0  AND  M  PARAMETERS 
BASELINE  LCOM  TO  1977  ACTUALS  FOR  SEYMOUR-JOHNSON  AFB 
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AVERAGE  PERCENT  DIFFERENCE  -  9.57 


TABLE  14  COMPARISON  OF  CRITICAL  KC-135A  0  AND  M  PARAMETERS 
BASELINE  LCOM  TO  1977  ACTUALS  FOR  CASTLE  AFB 


AVERAGE  PERCENT  DIFFERENCE  -  14.08 


(5)  Compare  selected  critical  O&M  output  parameters  from 
each  base's  baseline  runs  with  actual  1977  O&M  data  taken  from  the 
G033B  and  D056E  data  systems.  This  source  data  used  are  recorded  in 
Appendix  B.  The  difference  analyses  for  Loring,  Seymour-Johnson, 
and  Castle  AFB's  appear  in  Tables  12,  13  and  14  respectively. 

The  comparative  analyses  of  the  outputs  of  the  standard  and 
metrics  simulation  runs  against  the  baseline  runs  checked  the  success 
of  the  new  metrics  in  simulating  base-specific  situations.  The  overall 
findings  of  these  analyses  indicated  that  the  newly  developed  mainte¬ 
nance  metrics  were  approximately  equal  to  the  ASD  developed  standard 
KC-135A  metrics  in  producing  simulation  results  similar  to  the  base- 
specific  metrics  used  in  the  baseline  runs.  Both  types  produced 
simulated  outputs  that  were  generally  within  3%  of  the  baseline 
outputs  for  Loring  and  Castle  AFB's,  and  within  9%  for  Seymour-Johnson 
AFB.  These  deviations  were  considered  well  within  the  range  of 
acceptability  for  most  applications  of  the  KC-135A  LCOM  simulation. 

The  comparisons  of  the  outputs  of  the  baseline  simulation 
runs  with  actual  1977  O&M  histories  at  the  subject  bases  measured 
the  overall  fidelity  of  the  KC-135A  LCOM  with  the  ASD  standard  input 
module  (except  for  F-clock  values)  in  reproducing  actual  base  condi¬ 
tions.  These  comparisons  indicated  acceptable  levels  of  deviation 
between  the  LCOM  outputs  and  actual  1977  field  data.  The  average 
deviations  of  the  selected  O&M  parameters  were  under  10%  for  Loring 
and  Seymour-Johnson  AFB's  (see  Tables  12  and  13),  and  under  15%  for 
Castle  AFB  (see  Table  14). 

Since  the  results  of  the  Validation  Experiment  1  runs  as 
discussed  above  showed  such  low  deviations,  the  optional  follow-on 
experiments  shown  on  the  validation  plans  of  Figures  6,  7,  and  8  were 
not  performed. 


Ill  -  CONCLUSION 


1 .  SYNOPSIS 


This  report  describes  the  work  accomplished  under  Task  VIII 
of  an  eight  task  study  to:  "Develop  Maintenance  Metrics  To  Forecast 
Resource  Demands  Of  Weapon  Systems."  The  purpose  of  the  work  discussed 
in  this  interim  report  was  the  performance  of  LCOM  experiments  to  provide 
validation  evidence  for  the  maintenance  demand  prediction  metrics 
developed  during  the  previous  tasks.  This  validation  effort  was 
concluded  in  September  1980.  The  purpose  of  the  Task  VIII  validation 
effort  was  to  demonstrate  the  accuracy,  effects,  and  confidence  that 
users  of  the  developed  methodology  could  expect  when  using  the  new 
maintenance  metrics  in  place  of  existing  techniques  for  predicting 
equipment  maintenance  demand.  The  approach  to  this  portion  of  the 
study  effort  was  to  select  and  implement  existing  LCOM  simulations  of 
existing  aircraft;  and  to  conduct  calibrations  and  validation  experi¬ 
ments  which  simulated  existing  specific  basing  situations  of  the  subject 
aircraft.  The  calibration  runs  used  existing  maintenance  demand  metrics 
and  the  experimental  runs  used  the  newly  developed  maintenance  demand 
metrics.  Outputs  of  the  calibrations  and  experiments  were  then  compared 
to  show  how  well  the  new  metrics  could  simulate  specific  existing 
conditions  compared  to  present  methods.  The  objective  of  this  valida¬ 
tion  effort  was  to  provide  evidence  of  the  credibility  and  worth  of  the 
new  metrics  to  potential  users.  The  aircraft/basing  situations  chosen 
for  validation  were  the  F-15A/Bitburg  Air  Base  which  was  included  in 
the  original  study  data  base;  and  the  KC-1 35A/Loring  AFB,  KC-1 35A/Seymour- 
Johnson  AFB,  and  KC-1 35A/Castle  AFB  which  were  outside  the  original 
study  data  base. 

Results  of  the  work  accomplished  during  the  Task  VIII  effort 
and  included  in  this  report  are:  1)  performance  of  15  simulation  runs 
with  the  F-15A/Bitburg  model  using  various  combinations  of  the  existing 
metrics  and  the  newly  developed  metrics;  2)  comparative  analyses  among 
these  runs  to  indicate  the  credibility  and  acceptability  of  the  new 
metrics;  3)  performance  of  9  simulation  runs  with  the  KC-1 35A/ Lori ng 
model  as  in  1)  above;  4)  difference  analyses  of  the  outputs  of  these 
runs  as  in  2)  above;  5)  performance  of  9  simulation  runs  and  difference 
analyses  as  in  3)  and  4)  above  with  the  KC-135A/Seymour- Johnson  model; 
and  finally  6)  performance  of  simulation  runs  and  difference  analyses 
as  in  3)  and  4)  above  with  the  KC-135A/Castle  model.  The  newly  developed 
metrics  were  found  acceptable  in  all  cases  except  for  the  maintenance 
demand  metric  associated  with  the  F-15A  propulsion  system.  The  large 
deviation  in  simulation  results  caused  by  this  metric  indicates  the 
need  for  its  further  modification  and/or  refinement. 
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2. 


PROBLEMS  ASSUMPTIONS,  AND  UNCERTAINTIES 


No  significant  operational  problems  were  encountered  during 
work  on  Task  VIII.  The  usual  difficulties  and  debugging  requirements 
associated  with  new  applications  of  large  scale  computer  programs 
were  overcome  with  much  appreciated  aid  and  cooperation  of  the  HRL 
and  ASD  personnel  concerned.  All  intended  work  was  accomplished 
within  the  time  and  resources  allotted  for  this  portion  of  the  study. 

The  major  assumption  underlying  the  approach  to  the  valida¬ 
tion  effort  was  that  the  capability  of  the  new  maintenance  demand 
metrics  to  provide  an  adequate  portrayal  of  actual  aircraft/base 
operational  and  maintenance  situations  within  the  context  of  the 
LCOM  simulation  is  direct  evidence  of  the  accuracy  and  credibility 
of  the  maintenance  demand  predictions  of  the  newly  developed  mainte¬ 
nance  metric  models. 

The  greatest  uncertainty  arising  from  the  validation 
experiments  is  the  uncertainty  of  whether  enough  simulation  runs  were 
performed  in  each  experimental  configuration  to  provide  assurance 
that  random  variations  in  simulation  results  were  averaged  out. 

3.  FINDINGS  AND  RECOMMENDATIONS 


The  simulation  experiments  accomplished  during  the  Task 
VIII  effort  provided  the  following  overall  findings: 

(1)  For  the  F-15A/Bitburg  LCOM  experiments,  the  average 
deviation  between  the  baseline  simulation  and  the  simulations  using 
the  new  metrics  for  the  25  critical  output  variables  was  8.25%  for 
the  eleven  avionics  systems  and  61.83%  for  the  propulsion  system. 

(2)  For  the  KC-1 35A/Loring  LCOM  experiments,  the  average 
deviation  between  the  baseline  simulations  and  the  new  metrics 
simulations  for  the  25  critical  outputs  was  2.85%  for  all  thirty 
aircraft  systems  studied;  for  the  KC-135A/Seymour-Johnson  experiments, 
the  average  deviation  was  8.93%;  and  the  the  KC-135A/Castle  experi¬ 
ments,  the  average  deviation  was  2.79%. 

(3)  The  overall  fidelity  of  the  KC-135A  LCOM  was  indicated 
by  the  following  results  of  comparative  analyses  of  the  average 
deviations  between  the  simulated  outputs  of  the  baseline  model  runs 
and  1977  actua.  historical  data  for  seven  critical  O&M  parameters: 

KC-1 35A/LO ring  =  7.45% 

KC-135A/Seymour-Johnson  =  9.57% 

KC-1 35A/Castle  =  14.08% 
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The  above  findings  provide  evidence  that  the  new  maintenance 
metrics  in  their  present  stage  of  development  will  produce  adequate 
predictions  of  equipment  maintenance  demand  for  all  of  the  aircraft 
systems  studied,  except  propulsion,  under  diverse  design,  operational, 
and  environmental  conditions. 

Pursuant  to  the  above  findings  from  the  Task  VIII  valida¬ 
tion  effort,  it  is  concluded  that  the  newly  developed  maintenance 
metrics  for  all  but  the  propulsion  system  can  be  used  with  some 
confidence  to  predict  maintenance  demands  for  emerging  aircraft 
systems  and/or  new  basing  conditions.  It  is  recommended  that  the 
propulsion  maintenance  metric  be  used  with  cauticn  until  it  is 
further  investigated  and  refined.  It  is  further  recommended  that  a 
follow-on  study  be  implemented  to  refine  all  of  the  developed  metrics 
to  provide  "by-aircraft- type"  maintenance  demand  predictors  for 
even  better  credibility  and  data  fit.  The  metrics  developed  by  this 
study  were  derived  from  a  population  of  aircraft  which  included 
examples  of  all  types  in  Air  Force  inventory  (bombers,  fighters, 
transports,  and  trainers)  and  so  are  of  a  general  nature.  If  these 
metrics  were  expanded  into  separate  sets  to  be  specifically  applicable 
to  bombers,  fighters,  transports,  and  trainers,  higher  statistical 
confidence  could  be  placed  in  their  output  predictions. 
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GLOSSARY  OF  ABBREVIATIONS 


A8 

Air  Base 

Acft 

Aircraft 

AFB 

Air  Force  Base 

AFLC 

Air  Force  Logistics  Command 

AFM 

Air  Force  Manual 

AMAD 

Actual  Maintenance  Action  Demand 

ASD 

Advanced  Systems  Division 

BMW 

Bomb  Wing 

EMAD 

Estimated  Maintenance  Action  Demand 

F-Clock 

Failure  Clock 

FTW 

Fighter  Training  Wing 

Gen 

Generator 

IFF 

Identification  Friend  or  Foe 

Indie 

Indicator 

I/O 

Input/ Output 

LOOM 

Logistics  Composite  Model 

LOX 

Liquid  Oxygen 

MAC 

Military  Airlift  Command 

MAD 

Maintenance  Action  Demand 

Maint 

Maintenance 

MAW 

Military  Airlift  Wing 

NAV 

Navigation 

O&S 

Operations  and  Support 

PS/EAC 

Product  Support/Experience  Analysis  Center 

Ref 

Reference 

SAC 

Strategic  Air  Command 

TFW 

Tactical  Fighter  Wing 

TR 

Technical  Report 

TTW 

Tactical  Training  Wing 

UHF 

Ultra  High  Frequency 

WUC 

Work  Unit  Code 
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APPENDIX  A 


F-15A/BITBURG  LCOM 

FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 


TABLE  A1  Calculation  of  Actual  Maintenance  Action  Demands 
TABLE  A2  Calculation  of  Estimated  Maintenance  Action  Demands 
TABLE  A3  Calculation  of  Metric-Model -Adjusted  F-Clock  Values 
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TABLE  A1  CALCULATION  OF  ACTUAL  MAINTENANCE  ACTION  DEMANDS  ( F- 1 5A/BITBURG) 
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*  No  data  from  Bitburg  used  value  of  A06  from  F-l 5A * s  at  Luke  AFB. 

'■*  Operating  point  from  Bitburg  in  indeterminate  region  of  estimating  model. 
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Data 
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APPENDIX  C 

KC-135A/L0RING  AFB,  SEYMOUR- JOHNSON  AFB,  AND  CASTLE  AFB 
BASELINE  LCOM  FAILURE  CLOCK  WORKSHEETS 


TABLE  C1-I--C1-2  Baseline  LCOM  F-Clock  Calculations  for 
Loring  AFB 

TABLE  C2-1— C2-2  Baseline  LCOM  F-Clock  Calculations  for 
Seymour-Johnson  AFB 


TABLE  C3-1-C3-2 


Baseline  LCOM  F-Clock  Calculations  for 
Castle  AFB 
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TABLE  Cl-1 

KC-135A  BASELINE  LCOM  F-CLOCK  CALCULATIONS  FOR  LORIN6  AF3 


EQUIPMENT  SUBSYSTEM 

WUC 

F-  CLOCK 

TOTAL 

TOTAL  MAD 

BASELINE 

ID 

SORTIES 

PER  BASE 

F-CLOCK 

PER  BASE 

PER  YEAR 

VALUE 

PER  YEAR 

(1977) 

(SORTIES/ 

(1977) 

-T- 

_  MA) 

PROPULSION 

23A 

FA23AS 

1579 

41 

38.5 

789.5 

23B 

FA23BS 

29.3 

23C 

FA23CS 

17.5 

23D 

FA23DS 

37 

42.7 

23E 

FA23ES 

49 

32.2 

23H 

FA23HS 

152 

10.4 

23J 

FA23JS 

311 

5.1 

2 

789.5 

23K 

FA23KS 

258 

6.1 

23L 

FA23LS 

212 

7.4 

23M 

FA23MS 

246 

6.4 

23N 

FA23NS 

143 

11.0 

23P 

FA23PS 

320 

4.9 

23R 

FA23RS 

190 

8.3 

FA23R0 

32 

49.3 

230 

FA230S 

7 

225.6 

FLIGHT  INDICATORS 

511 

FA511S 

1579 

143 

11.0 

AIR  DATA  SYSTEM 

51B 

FA51BS 

157S 

77 

20.5 

H0RI2.  SITU.  INDIC. 

S1A 

FA51AS 

1579 

211 

7.5 

AUTOPILOT 

521 

FA521S 

1579 

mgm 

27.2 

FA5210 

9.1 

UHF  COMM.  SET 

63A 

FA63RS 

1579 

362 

4.4 

63R 

IFF  SET 

65B 

FA65BS 

1579 

92 

17.2 

INS  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

INSTR.  LANDING  SET 

71B 

FA71BS 

1579 

73 

21.6 

TACAN  SET 

71C 

FA71CS 

1579 

212 

7.4 

A-H  REF.  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

RADAR  SET 

72B 

FA72BS 

1579 

549 

2.3 

FUSELAGE 

111 

FA1115 

1579 

373 

mm 

RAOOME 

U1J 

WINDSHIELD 

114H 

mm 
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TABLE  Cl-2 

KC-135A  BASELINE  LCOM  F-CLOCK  CALCULATIONS  FOR  LOSING  AFB 


EQUIPMENT  SUBSYSTEM 

WUC 

F-CLOCK 

ID 

TOTAL 

SORTIES 

PER  BASE 
PER  YEAR 
(1977) 

TOTAL  MAD 
PER  BASE 
PER  YEAR 
(1977) 

BASELINE 

F-CLOCK 

VALUE 

(SORTIES/ 

=  MA) 

WINGS 

mmm 

FA11A0 

1579 

75 

21.1 

FA11JO 

204 

7.7 

lift  r^^B 

FA11KO 

197 

8.0 

FA116S 

12 

131.6 

FA1160 

24 

65.8 

117 

FA117S 

13 

121.5 

FA1170 

38 

41.6 

COCKPIT  FURNISHINGS 

12A 

FA12AS 

1579 

19 

FA12A0 

10 

ExB 

MAIN  LANDING  GEAR 

13A 

FA13A0 

1579 

525 

3.0 

BRAKES 

13C 

FA13CS 

1579 

337 

4.7 

STABILATOR 

11G 

FA11GO 

1579 

67 

23.6 

RUDDER 

14B 

FA14B0 

1579 

177 

8.9 

FLAPS 

14E 

FA14E0 

1579 

520 

3.0 

ENVIRON.  CONT.  SYS. 

412 

■TfTf 

1579 

*1 

38.5 

28 

56.4 

ELECT.  PWR.  GEN. 

421 

FA421S 

1579 

355 

4.4 

EXTERIOR  LIGHTS 

442 

(NOT  IN 

MODEL) 

HYDR.  PWR.  SYS. 

451 

FA451S 

1579 

501 

3.2 

INTERNAL  FUEL  TANKS 

MAIN  TANKS  1  S  2 

461 

FA461S 

1579 

132 

12.0 

MAIN  TANKS  3  &  4 

462 

FA4620 

148 

10.7 

L&R  WING  CNTR.  TANKS 

463 

FA4630 

70 

22.6 

LIQUID  OXYGEN  SYS. 

inm 

FA471S 

1579 

118 

13.4 

OXYGEN  REG 

LOX  CONVERTER 

47111 

FIRE  DETECTION 

494 

FA494S 

1579 

124 

12.7 

TABLE  C2-1 

KC-135A  BASELINE  LCOM  F-CLOCK  CALCULATIONS  FOR  SEYMOUR  JOHNSON  AFB 


EQUIPMENT  SUBSYSTEM 

WUC 

F-CLOCK 

ID 

TOTAL 

SORTIES 

PER  BASE 
PER  YEAR 
(1977) 

TOTAL  MAD 
PER  BASE 

PER  YEAR 
(1977) 

§H 

PROPULSION 

23A 

FA23AS 

782 

27 

29.0 

FA23A0 

1 

782.0 

23B 

FA23BS 

124 

6.3 

23C 

FA23CS 

57 

11.7 

23D 

FA23DS 

13 

60.2 

23E 

FA23ES 

75 

10.4 

23H 

FA23HS 

169 

4.6 

23  J 

FA23JS 

294 

2.7 

FA23J0 

2 

391.0 

23K 

FA23KS 

213 

3.7 

23L 

FA23LS 

108 

7.2 

23M 

FA23MS 

149 

5.2 

23N 

FA23NS 

22 

35.5 

23P 

FA23PS 

199 

3.9 

23R 

FA23RS 

137 

5.7 

FA23R0 

23 

34.0 

230 

FA230S 

74 

10.6 

FLIGHT  INDICATORS 

511 

FA511S 

782 

103 

7.6 

AIR  DATA  SYSTEM 

51B 

FA51BS 

782 

62 

12.6 

HORIZ.  SITU.  INDIC. 

51A 

FA51AS 

782 

124 

6.3 

AUTOPILOT 

521 

FA521S 

782 

19 

41.2 

FA5210 

58 

13.5 

UHF  COMH.  SET 

63A 

FA63RS 

782 

102 

63R 

I 

IFF  SET 

65B 

FA65BS 

782 

69 

11.3 

INS  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

INSTR.  LANOING  SET 

71B 

FA718S 

782 

19 

41.2 

TACAN  SET 

71C 

FA71CS 

782 

131 

6.0 

A-H  REF.  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

RADAR  SET 

72B 

FA72BS 

782 

351 

2.2 

FUSELAGE 

111 

FA111S 

782 

117 

RADOME 

111J 

WINDSHIELD 

114H 

■ 

C-4 

D1 94-1 0089-4 


'1 


t 


TABLE  C2-2 

KC-135A  BASELINE  LCOM  F-CLOCK  CALCULATIONS  FOR  SEYMOUR  JOHNSON  AFB 


EQUIPMENT  SUBSYSTEM 

wuc 

F-CLOCK 

ID 

TOTAL 

SORTIES 

PER  BASE 

PER  YEAR 
(1977) 

TOTAL  MAD 
PER  BASE 

PER  YEAR 
(1977) 

BASELINE 

F-CLOCK 

VALUE 

(SORTIES/ 

_  MA) 

MINGS 

11A 

F All AO 

782 

46 

17.0 

11J 

FA11JO 

255 

3.1 

1 1 K 

FA11KO 

278 

2.8 

116 

FA116S 

6 

130.3 

FA1160 

11 

71.1 

117 

FA117S 

5 

156.4 

FA1170 

14 

55.9 

COCKPIT  FURNISHINGS 

12A 

FA12AS 

782 

11 

71.1 

FA12AO 

5 

156.4 

MAIN  LANDING  GEAR 

13A 

FA13AO 

782 

239 

3.3 

BRAKES 

13C 

FA13CS 

782 

73 

10.7 

STABILATOR 

11G 

FA11GO 

782 

13 

60.2 

RUDDER 

148 

FA14B0 

782 

32 

24.4 

FLAPS 

14E 

FA14E0 

782 

164 

4.8 

ENVIRON.  CONT.  SYS. 

412 

FA412S 

782 

24 

32.6 

FA4120 

16 

48.9 

ELECT.  PWR.  GEN. 

421 

FA421S 

782 

276 

2.8 

EXTERIOR  LIGHTS 

442 

(NOT  IN 

MODEL) 

HYOR.  PWR.  SYS. 

451 

FA451S 

782 

139 

5.6 

INTERNAL  FUEL  TANKS 

MAIN  TANKS  1  &  2 

461 

FA461S 

782 

25 

31.3 

MAIN  TANKS  3  &  4 

462 

FA4620 

20 

39.1 

L&R  WING  CNTR.  TANKS 

463 

FA4630 

11 

71.1 

LIQUID  OXYGEN  SYS. 

471 

FA471S 

782 

69 

11.3 

OXYGEN  REG 

47131 

LOX  CONVERTER 

47111 

FIRE  DETECTION 

494 

FA494S 

782 

69 

11.3 

TABLE  C3-1 

KC-135A  BASELINE  LCOM  F-CLOCK  CALCULATIONS  FOR  CASTLE  AFB 


EQUIPMENT  SUBSYSTEM 

WUC 

F-CLOCK 

ID 

TOTAL 

SORTIES 

PER  BASE 

PER  YEAR 
(1977) 

TOTAL  MAD 
PER  BASE 
PER  YEAR 
(1977) 

BASELINE 

F-CLOCK 

VALUE 

(SORTIES/ 

_  MA) 

PROPULSION 

23A 

FA23AS 

3043 

107 

28.4 

FA23A0 

5 

608.6 

23B 

FA23BS 

375 

8.1 

23C 

FA23CS 

164 

18.6 

23D 

FA23DS 

58 

52.5 

23E 

FA23ES 

329 

9.2 

23H 

FA23HS 

384 

7.9 

23J 

FA23JS 

641 

4.7 

FA23J0 

5 

608.6 

23  K 

FA23KS 

500 

6.1 

23L 

FA23LS 

325 

9.4 

23M 

FA23MS 

832 

3.7 

23N 

FA23NS 

270 

11.3 

23P 

FA23PS 

584 

5.2 

23R 

FA23RS 

436 

7.0 

FA23R0 

72 

42.3 

230 

FA230S 

60 

50.7 

FLIGHT  INDICATORS 

511 

FA511S 

3043 

400 

7.6 

AIR  DATA  SYSTEM 

SIB 

FA518S 

3043 

221 

13. *3 

HORIZ.  SITU.  INOIC. 

51A 

FA51AS 

3043 

736 

4.1 

AUTOPILOT 

521 

FA521S 

3043 

116 

26.2 

FA5210 

347 

8.8 

UHF  COMM.  SET 

63A 

FA63RS 

3043 

390 

7.8 

63R 

IFF  SET 

65B 

FA65BS 

3043 

199 

15.3 

INS  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

INSTR.  LANDING  SET 

71B 

FA71BS 

3043 

111 

27.4 

TACAN  SET 

71C 

FA71CS 

3043 

296 

10.3 

A-H  REF.  SET 

(NOT 

INSTALLED 

IN  SAMPLE 

KC-135A) 

RADAR  SET 

723 

FA72BS 

3043 

1048 

2.9 

FUSELAGE 

111 

FA111S 

3043 

417 

7.3 

RADOME 

1110 

WINDSHIELD 

114H 

C-6 
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TABLE  C3-2 

KC-135A  BASELINE  LCQM  F-CLOCK  CALCULATIONS  FOR  CASTLE  AFB 


EQUIPMENT  SUBSYSTEM 

WUC 

F-CLOCK 

TOTAL 

TOTAL  MAD 

BASELINE 

ID 

SORTIES 

PER  BASE 

F-CLOCK 

PER  BASE 

PER  YEAR 

VALUE 

PER  YEAR 

(1977) 

(SORTIES/ 

(1977) 

4- 

_  MA) 

MINGS 

11A 

FA11A0 

3043 

162 

18.8 

11J 

FA11J0 

341 

8.9 

1  IK 

FA11K0 

256 

11.9 

116 

FA116S 

35 

86.9 

FA1160 

70 

43.5 

117 

FA117S 

21 

144.9 

FA1170 

63 

48.3 

COCKPIT  FURNISHINGS 

12A 

FA12AS 

3043 

22 

138.3 

FA12A0 

11 

276.6 

MAIN  LANDING  GEAR 

13A 

FA13A0 

3043 

1232 

2.5 

BRAKES 

13C 

FA13CS 

3043 

566 

5.4 

STABILATOR 

11G 

FA11F0 

3043 

57 

53.4 

RUDDER 

14B 

FA14B0 

3043 

215 

14.2 

FLAPS 

14E 

FA14E0 

3043 

509 

6.0 

ENVIRON.  CONT.  SYS. 

412 

FA412S 

3043 

98 

31.1 

FA4120 

66 

46.1 

ELECT.  PWR.  GEN. 

421 

FA421S 

3043 

732 

4.2 

EXTERIOR  LIGHTS 

442 

(NOT  IN 

MODEL) 

HYDR.  PWR.  SYS. 

451 

FA451S 

3043 

822 

3.7 

INTERNAL  FUEL  TANKS 

MAIN  TANKS  I  &  2 

461 

FA461S 

3043 

209 

14.6 

MAIN  TANKS  3  &  4 

462 

FA4620 

239 

12.7 

L&R  WING  CNTR.  TANKS 

463 

FA4630 

173 

17.6 

LIQUID  OXYGEN  SYS 

471 

FA471S 

3043 

211 

14.4 

OXYGEN  REG. 

47131 

LOX  CONVERTER 

47111 

FIRE  DETECTION 

494 

FA494S 

3043 

419 

7.3 
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APPENDIX  D 


KC-1 35A/L0RING  LOOM 

FAILURE  CLOCK  TRANSFORMATION  WORKSHEETS 


TABLE  DL1-1 — 4  Calculation  of  Actual  Maintenance  Action  Demands 

TABLE  DL2-1 — 4  Calculation  of  Estimated  Maintenance  Action  Demands 
and  Metrics-Model -Adjusted  F-Clock  Values 


KC-1 35A/SEYM0UR-J0HNS0N  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEETS 


TABLE  DS1-1 — 4  Calculation  of  Actual  Maintenance  Action  Demands 

TABLE  DS2- 1  — 4  Calculation  of  Estimated  Maintenance  Action  Demands 
and  Metrics-Model -Adjusted  F-Clock  Values 


KC-1 35A/ CASTLE  LCOM 

FAILURE  CLOCK  TRANSFORMATION  WORKSHEETS 


TABLE  DC1-1 — 4  Calculation  of  Actual  Maintenance  Action  Demands 

TABLE  DC2-1 — 4  Calculation  of  Estimated  Maintenance  Action  Demands 
and  Metrics-Model -Adjusted  F-Clock  Values 
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KC-135A/LORING  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 
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KC-135A/L0RING  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 


KC-135A/L0RING  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 


KC-135A/L0RING  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 

TABLE  DL2-1  CALCULATION  OF  ESTIMATED  MAINTENANCE  ACTION  DEMANDS 
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KC-135A/L0RING  AFB  LOOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 

TABLE  DL2-3  CALCULATION  OF  ESTIMATED  MAINTENANCE  ACTION  DEMANOS 
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KC- 1 35A/LOR ING  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 


KC-135A/SEYM0UR  JOHNSON  AFB  LCOM 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 


KC-135A/SEYH0UR  JOHNSON  AFB  LCOH 
FAILURE  CLOCK  TRANSFORMATION  WORKSHEET 
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TABLE  E8  LC-135A/ CASTLE  AFB  SIMULATION  RESULTS,  ASD  STANDARD  F-CLOCKS 
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TABLE  EIO  KC-I35A/CASTLE  AFB  SIMULATION  RESULTS,  1977  BASELINE  F-CLOCKS 
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